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SILVER  BULLET  OR  URBAN  LEGEND? 


■  SILVER  BULLET  OR  URBAN  LEGEND? 

>  ANTECDOTES 

>  ROI  DEFINED 

>  EVALUATION  LEVELS 


PERFORMANCE? 

(DATA) 
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SILVER  BULLET  OR  URBAN  LEGEND? 

ANTECDOTES 


“The  biggest  value  that  Training  ROI  ever  produced  was  to  sell  books  for  a  few  authors,  so  unless  you  are 
one  of  those  authors,  forget  all  you  have  heard  and  read  about.  Training  ROI  is  perhaps  the  least 
meaningful  metric  that  you  can  determine  when  it  comes  to  training,  rated  just  below  “smile”  sheets.  It  is 
NOT  indicative  of  the  contribution  that  training  makes  to  an  organization.” 

Bob  Dust,  GM  Gyro  Systems  Inc.,  “The  Myth  of  Training  ROI,”  Jan  ‘04 


The  problem  is  that  nobody  is  quite  sure  what  the  appropriate  metrics  are  for  measuring  ROI  for  learning.  Is 
it  student  throughput  or  time  to  mastery?  Is  it  dropout  rates  or  full-time  equivalents  returned  to  the 
workforce?  One  thing  is  certain:  many  smart  people  are  completely  befuddled  by  the  topic.” 

Marcia  L.  Conner,  Learnativity.com,  “How  do  I  measure  return  on  investment  (ROI)  for  my  learning  program?,  5  Apr  ‘02" 


“To  some  people--me  included--the  traditional  concept  of  training  ROI  is  obsolete.  Astute  training  managers 
employ  business  metrics,  not  evaluation  levels,  I  believe.  Business  unit  managers  value  time  more  than 
ROI.  Major  decisions  are  based  on  descriptive  business  cases,  not  pro  forma  budgets.  Senior  executives 
tend  to  be  more  interested  in  the  top  line  (dramatic  growth  from  new  markets  and  innovation)  than  the 
bottom  line  (the  accounting  fiction  of  profits).” 

Jay  Cross,  CEO  of  Internet  Time  Group,  “A  Fresh  Look  at  ROI,”  Jan  01 


“The  reason  why  I  won't  do  an  ROI  study  is  that  any  major  change  effort  within  an  organization  requires  not 
just  training,  but  many  other  factors.” 

Daniel  R.  Tobin,  PhD,  Corporate  learning  Strategies,  “The  Fallacy  of  ROI  Calculations,  ‘98 
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SILVER  BULLET  OR  URBAN  LEGEND? 

ROI  DEFINED 


BCR  =  PROGRAM  BENEFITS 

PROGRAM  COSTS 

BCR:  BENEFITS  COST  RATIO 


ROI%  =  NET  PROGRAM  BENEFITS  x  100 

PROGRAM  COSTS 

ROI:  RETURN  ON  INVESTMENT 


Phillips  and  Stone.  How  To  Measure  Training  Results:  A  Practical  Guide  To  Tracking  The  Six  Key  Indicators.  2002,  page  11, 19 
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SILVER  BULLET  OR  URBAN  LEGEND? 

EVALUATION  LEVELS 


PHILLIPS’ 
FIFTH  LEVEL 


LEVEL  3 
BEHAVIOR 


LEVEL  2 
LEARNING 


LEVEL  1 
CUSTOMER  REACTION 


KIRKPATRICK’S  FOUR  LEVELS  OF  EVALUATION 


DATA 

VALUE 

CUSTOMER 

FOCUS 

FREQUENCY 
OF  USE 

DATA 

COLLECTION 

HIGHEST 

CLIENT 

LEAST 

HARDEST 

▲ 

▲ 

▲ 

i 

▼ 

▼ 

T 

T 

LOWEST 

CONSUMER 

MOST 

EASIEST 

TRAINING  EVALUATION  LEVEL  ATTRIBUTES 


Phillips.  The  Bottom  Line  on  ROI:  Basics.  Benefits.  &  Barriers  to  Measuuinq  Training  and  Performance  Improvement.  2002,  page  45 
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EVALUATION  FRAMEWORKS 


■  EVALUATION  FRAMEWORKS 

>  ANALYSIS  OF  ALTERNATIVES 

>  TRAINING  SYSTEMS  REQUIREMENTS  ANALYSIS 

>  M&S  RETURN  ON  INVESTMENT 

>  TRAINING  RETURN  ON  INVESTMENT 

>  THE  HYBRID  MODEL 
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EVALUATION  FRAMEWORKS 

ANALYSIS  OF  ALTERNATIVES  (AOA) 
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EVALUATION  FRAMEWORKS 

TRAINING  SYSTEMS  REQUIREMENTS  ANALYSIS  (TSRA) 


TRAINING  SITUATION  ANALYSIS  (TSA) 


EXISTING  SKILLS/KNOWLEDGE 
REQUIRED  SKILLS/KNOWLEDGE 

I 


TRAINING  ANALYSIS 
TASK  &  SKILLS  ANALYSIS 


(1) 

(2) 

DETERMINE 

DETERMINE 

TRAINING 

TRAINING 

REQUIREMENT 

OBJECTIVE 

TSA  Doc 


Training  System  Plan 


Engineering  Specs 


TRAINING  SYSTEM  FUNCTIONAL  DOCUMENT  (TSFD) 


TSRD  Doc 


(5)  I4J 

DETERMINE 
FUNCTIONAL 
DESCRIPTION 


TRAINING  SYSTEMS  ALTERNATIVES  REPORT  (TSAR) 


(3) 

EVALUATE 

ALTERNATIVE 


COST  &  RESOURCE  CONSTRAINTS 
TRAINING  EFFECTIVENESS  ANALYSIS 
TRAINING  TECHNOLOGY  ASSESSMENT 
MEDIA  ANALYSIS 


E 

CLASSROOM  MEDIA  TRAINING 

E 

INTERACTIVE  MULTI-MEDIA  INSTRUCTION  TRAINING 

E 

SIMULATION/STIMULATION  TRAINING 

E 

DISTRIBUTED  LEARNING 

E 

TECHNICAL  TRAINING  EQUIPMENT  TRAINING 

E 

EMBEDDED  TRAINING 

E 

NONE 

CRITERIA 

COST  VS.  BENEFIT 
TRAINING  EFFECTIVENESS 
USER  ACCEPTANCE 
RETURN  ON  INVESTMENT 


TSAR  Doc 


TRAINING  DEVISE  REQUIREMENTS  DOCUMENT  (TSRD) 


(4) 

SPONSOR 

DECISION 


USER  ACCEPTANCE 


TSRD  Doc 


NAWC  Training  Systems  Division.  Training  Systems  Requirements  Analysis  Webpage,  30  August  2007 
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EVALUATION  FRAMEWORKS 

M&S  RETURN  ON  INVESTMENT 
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EVALUATION  FRAMEWORKS 

TRAINING  RETURN  ON  INVESTMENT  (ROI) 
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EVALUATION  FRAMEWORKS 

HYBRID 


UNCLASSIFIED 


SOAL 


12 
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COURSES  OF  ACTION  (ALTERNATIVES) 


■  COURSE  OF  ACTION  (ALTERNATIVES) 

>  TRAINING  SPECTRUM 

>  AC-130U  ILLUSTRATION 
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COURSES  OF  ACTION 

TRAINING  SPECTRUM 


UNCLASSIFIED 


I 

COST 


ALTERNATIVES? 

>  MATERIAL 

>  NEW 

>  SYSTEMS 

>  SOFTWARE 

>  COURSWARE 

>  MODIFICATIONS 

>  OBSOLESCENCE 

>  CONCURRENCY 

>  NON-MATERIEL 


CONSTRUCTIVE  VIRTUAL  LIVE 


real  people  trained  by 
providing  inputs  to 
simulated  systems 
(machine-to-machine) 


real  people  training  by 
operating 
simulated  systems 


real  people  trained  by 
operating 
real  systems 


EFFECTIVE 


AFAMS.  Return  on  Investment  oF  Modeling  and  Simulation  (M&S)  Workshop  Briefing.  April  2008 
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COURSES  OF  ACTION 

AC-130U  ILLUSTRATION 


UNCLASSIFIED 


WHY  NOT  LIVE  TRAINING? 

>  SAFETY 

>  SECURITY 

>  SCENARIOS 

>  COST 


COST 
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D 

CLASSROOM/  DESK  PART  FLIGHT  FULL 

COURSEWARE  TOP  TASK  TRAINING  MOTION 

INSTRUCTION  TRAINERS  TRAINERS  DEVICES  SIMULATORS 


EFFECTIVE 
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EFFECTIVENESS  EVALUATION 
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UNCLASSIFIED 

EFFECTIVENESS  EVALUATION 

ATTRIBUTES 


■  JOINT  TRAINING 

■  MODELING  &  SIMULATION 

> 

RIGHT  INFORMATION 

> 

REUSE 

> 

RIGHT  AUDIENCE 

> 

READINESS 

> 

ADAPTABILITY 

> 

EFFICIENCY 

> 

RESPONSIVENESS 

> 

RISK  REDUCTION 

> 

INTEROPERABILITY 

> 

EFFECTIVENESS 

> 

LEARNING  TRANSFERANCE 

> 

PERSISTENT/REINFORCED 

> 

ENVIRONMENT 

> 

NETWORKED 

> 

LIVES 

> 

EXPEDITIONARY 

> 

TIME 

> 

EFFICIENCY 

JCS.  Joint  Functional  Concept  for  Joint  Traininq,  Julv  2007 

AFAMS.  Return  on  Investment  oF  Modelinq  and  Simulation  (M&S)  Workshop  Briefinq.  April  2008 
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EFFECTIVENESS  EVALUATION 

METRICS 


OBJECTIVE  1  -  MISSION  IMPACT/OUTCOME 


LEVEL  3 
BEHAVIOR 


OBJECTIVE  2  -  TASK  APPLICATION/PERFORMANCE 


LEVEL  2 
LEARNING 


REACTION 


OBJECTIVE  3  -  LEARNING  ATTAINMENT 


OBJECTIVE  4  -  CUSTOMER  REACTION/SATISFACTION 


KIRKPATRICK’S  FOUR  LEVELS  OF  EVALUATION 


Phillips.  The  Bottom  Line  on  ROI:  Basics.  Benefits.  &  Barriers  to  Measuuinq  Training  and  Performance  Improvement.  2002,  page  45 
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UNCLASSIFIED 

EFFECTIVENESS  EVALUATION 

METRICS 


OBJECTIVE  1  - 

MISSION  IMPACT 

MEASURES  OF  EFFECTIVENESS 

DATA  SOURCES 

1-X:  MSN  EFFECTIVENESS 

■  ACTION/IMPROVEMENT  PLANS 

(OUTPUT-ORIENTED) 

■  ASSIGNMENTS  RELATED  TO  PROGRAM 

■  FOLLOW-UP 

e.g.:  SAFETY 

>  PROGRAM  SESSIONS 

ENVIROMENTAL  IMPACT 

>  QUESTIONNAIRES 

TARGET  NEUTRALIZED 

■  PERFORMANCE 

>  CONTRACTING 

>  MONITORING 

ISOLATING  TRAINING  EFFECTS  DIFFICULT 

Phillips  and  Stone.  How  To  Measure  Traininq  Results:  A  Practical  Guide  To  Trackinq  The  Six  Kev  Indicators.  2002,  paqe  34 
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UNCLASSIFIED 


EFFECTIVENESS  EVALUATION 

METRICS 


OBJECTIVE  2  - 

TASK  APPLICATION 

MEASURES  OF  EFFECTIVENESS 

DATA  SOURCES 

2-X:  TASK  EFFECTIVENESS 

■  ACTION/IMPROVEMENT  PLANS 

(INPUT  ORIENTED) 

■  ASSIGNMENTS  RELATED  TO  PROGRAM 

■  OBSERVATIONS  ON-THE-JOB 

e.g.:  TIME 

■  FOLLOW-UP 

RESOURCES 

>  FOCUS  GROUPS 

READINESS 

>  INTERVIEWS 

RISK  REDUCTION 

>  PROGRAM  SESSIONS 

>  QUESTIONNAIRES 

>  SURVEYS 

ISOLATING  TRAINING  EFFECTS  DIFFICULT 

■  PERFORMANCE 

>  CONTRACTING 

Phillips  and  Stone.  How  To  Measure  Traininq  Results:  A  Practical  Guide  To  Trackinq  The  Six  Kev  Indicators.  2002,  paqe  34 
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EFFECTIVENESS  EVALUATION 

METRICS 


OBJECTIVE  3  -  LEARNING 


MEASURES  OF  EFFECTIVENESS 


DATA  SOURCES 


3-1:  TRAINING  PROFICIENCY  RATIO 
3-2:  TRAINING  THROUGHPUT  RATIO 
3-3:  TRAINING  APPLICABILITY  RATIO 
3-4:  TRAINING  APPROPRIATENESS  RATIO 


■  ASSESSMENTS 

>  SELF 

>  TRAINER 

■  TESTS 

>  FORMAL 

>  CRITERIA  REFERENCED 

>  PERFORMANCE  BASED 

>  SIMULATION 

■  EXERCISES/ACTIVITIES 


Bob  Dust.  The  Mvth  of  Traininq  ROI.  January  2004 

Phillips  and  Stone.  How  To  Measure  Traininq  Results:  A  Practical  Guide  To  Trackinq  The  Six  Kev  Indicators.  2002,  paqe  34 
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EFFECTIVENESS  EVALUATION 

METRICS 


OBJECTIVE  4 

-  REACTION 

MEASURES  OF  EFFECTIVENESS 

DATA  SOURCES 

4-1:  STUDENT  COURSE  REACTION  RATINGS 

■  INITIAL 

4-1-1:  COURSE  OVERALL 

4-1-2:  COURSE  CONTENT 

4-1-3:  COURSE  METHODLOGY 

4-1-4:  COURSE  FACILITY 

4-1-5:  COURSE  INSTRUCTION 

4-1-6:  COURSE  CHANGE  ASSESSMENT 

>  INTERVIEWS 

>  QUESTIONNAIRES 

4-2:  ADMINSTRATION  REACTION  RATINGS 

4-1-1:  CURRICULUM  OVERALL 

4-1-2:  CURRICULUM  CONTENT 

4-1-3:  CURRICULUM  METHODLOGY 

4-1-4:  CURRICULUM  FACILITY 

4-1-5:  CURRICULUM  INSTRUCTION 

4-1-6:  CURRICULUM  CHANGE  ASSESSMENT 

Phillips  and  Stone.  How  To  Measure  Traininq  Results:  A  Practical  Guide  To  Trackinq  The  Six  Kev  Indicators.  2002,  paqe  34 
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EFFECTIVENESS  EVALUATION 

METRICS 


TRA IN  NG  ISOLATION/MONETARY  CONVERS  ON 


TRAINING  EFFECTS  ISOLATON 

■  CONTROL  GROUPS 

■  TREND  LINE  ANALYSIS 

■  FORECASTING 

■  PARTICIPANT  ESTIMATE 

■  SUPERVISOR  ESTIMATE 

■  MANAGEMENT  ESTIMATE 

■  CUSTOMER  INPUT 

■  EXPERT  ESTIMATE 

■  SUBORDINATE  INPUT 

■  OTHER  FACTORS  IMPACT 


TRAINING  “$”  CONVERSION 

■  CONVERTING 

>  OUTPUT  TO  CONTRIBUTION 

>  COST  OF  QUALITY 

>  EMPLOYEE  TIME 

■  HISTORICAL  COSTS 

■  INTERNAL/EXTERNAL  EXPERTS 

■  INTERNAL/EXTERNAL  DATABASES 

■  PARTICIPANTS’  ESTIMATES 

■  LINKING  WITH  OTHER  MEASURES 

■  SUPERVISOR/MANAGER  ESTIMATES 

■  TRAINING  STAFF  ESTIMATES 


Phillips  and  Stone.  How  To  Measure  Training  Results:  A  Practical  Guide  To  Tracking  The  Six  Key  Indicators.  2002,  page  34 
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COST  EVALUATION 

OVERVIEW 


UNCLASSIFIED 


■  COST  EVALUATION 

>  ATTRIBUTES 

>  METRICS 
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Cost 


COST  EVALUATION 

ATTRIBUTES 


UNCLASSIFIED 


GOAL:  CHANGE  =  REDUCED  LCC 


O&M 


Time 


4  .  »| 

Design^ - H 

Manufacture 

H - ► 

Logistics  Support 

◄ - ► 

LCC  =  Total  Ownership  Cost 


■  Research  and  Development  -  All  costs  required  to 
research  and  develop  a  system  before  committing  it  to 
production  (engineering  design,  manufacturing  of  test 
articles,  testing  to  prove  the  design,  contractor  ST&E, 
engineering  development  equipment,  software 
development,  intelligence  design,  data,  etc) 

■  Investment/Production  -  Costs  associated  with  the 
fabrication,  assembly  and  delivery  of  a  system  (prime 
mission  equipment,  support  equipment,  initial  spares, 
modification  to  existing  platforms,  intelligence  production, 
integration  costs,  data,  etc) 

■  Operations  &  Support  -  Includes  all  direct  and  indirect 
costs  required  for  operation  and  support  of  a  system  (cost 
of  personnel,  materials,  facilities,  depot  maintenance, 
inventory  management  control,  intelligence  support,  data, 
etc) 

■  Disposal  -  Cost  to  dispose  of  the  system  after  its  useful 
life  (disposal  or  long-term  storage  costs,  environmental  and 
related  costs,  development  and  manufacturing  cleanup 
costs,  etc) 
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COST  EVALUATION 

METRICS 


■  MEASURES  OF  COST 

>  BREAK  EVEN 

>  PAYBACK 
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COST-EFFECTIVENESS  EVALUATION 

OVERVIEW 


COST-EFFECTIVENESS  EVALUATION 

>  ATTRIBUTES 

>  METRICS 


A  EFFECTIVENESS 
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COST-EFFECTIVENESS  EVALUATION 

ATTRIBUTES 


PROBABILITY  OF  RISK 

Process  Past 

Observed  Process 

Level/ 

Performance 

The  process... 

Characteristics 

The  process  looks  like... 

Likelihood 

...Risk  events  are  imminent 
and  cannot  be  avoided  under 
current  conditions  - 
incapable  process 

Lack  of  planning  and 
management  for  any  process 
change;  any  success  is  due  to 
individual  effort/knowledge 

5 

Nearly 

Certain 

...expects  risk  events  and 
most  of  them  are  likely  to 
occur  -  incapable  process 

Process  management  based  on 
experience  with  similar 
processes;  successful  practices 
used 

4 

Highly 

Likely 

...anticipates  risk  events  but 
may  not  avoid  them  - 
marginally  capable  process 

Tech  and  management  aspects 
of  process  documented; 
standards  used;  sub-processes 
understood  and  used  to  improve 
process 

3 

Likely 

...has  usually  avoided  or 
resolved  risk  events  in  similar 
cases  -  capable  process 

Quantitative  management  of 
process  capability;  trends  are 
predicted 

2 

Unlikely 

...will  effectively  avoid  or 
resolve  risk  events  using 
standard  practices  -  highly 
capable  process 

Continuous  improvement  is 
norm;  best  practices  used; 
process  capability  expanded 
routinely 

1 

Remote 

M 

M 

n 

L 

M 

M 

L 

L 

M 

M 

L 

L 

L 

M 

M 

L 

L 

L 

L 

M 

1  2  3  4  5 

IMPACT 


IMPACT  OF  RISK 

Effectiveness 

Schedule 

Cost 

Level 

Minimal/No  Impact 

Minimal/No  Impact 

Minimal/No  Impact 

1 

Acceptable;  Some 

Margin  Reduction 

Meets  milestones 

Increase  <5% 

2 

Acceptable;  Significant 
Margin  Reduction 

Minor  milestone  slip 

Increase  5-7% 

3 

Acceptable,  No  Margin 
Remaining 

Major  milestone  slip 

Increase  >7-10% 

4 

Unacceptable 

Not  Meet  Milestones 

Increase  >10% 

5 
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UNCLASSIFIED 

COST-EFFECTIVENESS  EVALUATION 

METRICS 


UNCLASSIFIED 
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SUMMARY 


UNCLASSIFIED 


■  1.  BENCHMARK  CURRENT  TRAINING  PROGRAM 

>  TRAINING  PROGRAM  EFFECTIVENESS  (INCLUDE  CAPABILITY  GAPS) 

>  TRAINING  PROGRAM  COST 

■  2.  TREAT  TRAINING  AS  EXPENSE 

>  COMPARE  TRAINING  PROGRAM  COSTS  (CURRENT  vs  NEW) 

>  DETERMINE  PAYBACK  OF  NEW  TRAINING  PROGRAM,  IF  ANY 

■  3.  ISOLATE  THE  BENEFITS 

>  COMPARE  TRAINING  PROGRAM  EFFECTIVENESS  (CURRENT  vs.  NEW) 

>  DETERMINE  BENEFITS  OF  NEW  TRAINING  PROGRAM,  IF  ANY 

■  4.  AVOID  THE  “ROI”  RATIO 

>  ONE  NUMBER  MEANS  NOTHING 

>  CONSIDER  THE  FOLLOWING  EXPRESSIONS  (OBJ.  3)  OF  RETURN: 

-  _  %  IMPROVED  PROFICIENT 

-  _  %  IMPREOVED  THROUGHPUT 

-  _  %  INCREASES  SKILLS  VALUED  BY  COMMAND 

%  INCREASED  TRAINING  SPECIFIC  PEOPLE  IN  SPECIFIC  FUNCTIONS 
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UNCLASSIFIED 

QUESTIONS? 


DILBERT 


SV  SCOTT  POPfilS 


M Y  SIMULATOR  IS 
TOO  SLOU.  I  NEED 
TO  UPGRADE  IT. 


I  NEED  A  COST- 
BENEFIT  ANALYSIS 
INCLUDING  THE  COST 
OF  ALL  ALTERNATIVES. 
AND  VICE  PRESIDENT 
APPROVAL. 


IT  UIAS  EASTER  TO  GET 
A  SECOND  JOB  AND  PAY 
FOR  THE  UPGRADE 
PAYSELF. 
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